Excellent idea IMO! Does ORT give errors for license classifications that are used but not defined?

I also think this is an excellent idea.

Does ORT give errors for license classifications that are used but not defined?

Yes, that's why currently the CI check fails. But this actually is a blocker here: We'd not want to define a category for each different URL... not sure how to solve that yet.

If the check for unknown categories was "relegated" to Evaluator rules from ORT's license classification parsing logic, that would allow only checking for categories that aren't prefixed by url, for example. If possible, a low-friction way could be that the evaluator rules had a default logic that checks for unknown categories as it is know, but it could be overriden in evaluator.rules.kts. Do you think it would be possible?

Do you think it would be possible?

It's surely technically possible, but I believe there would be push-back from members of the ORT community for such a breaking change. I'll clarify on that.